Biological-information detecting device and biological-information detecting method

ABSTRACT

A biological-information detection device includes a video input section accepting video signals including three wavelength components in an infrared region of reflected light from an object, a wavelength detecting section acquiring a wavelength and an intensity of the reflected light from the video signals, a face feature amount detecting section detecting a plurality of feature points of a face based on the video signals, a measurement target area identifying section identifying a measurement target area on a basis of the plurality of feature points of the face detected, a wavelength fluctuation detecting section detecting a difference between a wavelength of reflected light from the measurement target area at a certain point in time and a wavelength of reflected light at a point in time preceding the certain point in time, and a pulse wave detecting section detecting a change in the detected difference according to the point in time.

CLAIM OF PRIORITY

The present application claims priority from Japanese patent applicationJP2018-205489 filed on Oct. 31, 2018, the content of which is herebyincorporated by reference into this application.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a technique for detecting biologicalinformation.

2. Description of the Related Art

As a method for acquiring biological information, a technique isavailable that allows real-time, non-contact detection using a microwaveor a camera. In particular, for pulse detection using a camera, a cameramodule has been increasingly miniaturized and mounted in portableterminals including smartphones. Such a technique has been increasinglyspreading.

As a technique for pulse detection using imaging, a method is availablethat includes tracing particularly a G signal included in an RGB signalin a face video to detect a pulse (Verkruysse, Wim, Lars O. Svaasand, J.Stuart Nelson, “Remote plethysmographic imaging using ambient light.”,Optics express 16. 26 (2008): 21434-21445).

To use the RGB signal and separate noise, another available method usesindependent component analysis to separate noise (Ming-Zher Poh, DanielJ. McDuff, Rosalind W. Picard, “Non-contact, automated cardiac pulsemeasurements using video imaging and blind source separation.”, OpticsExpress 18. 10 (2010): 10762-10774). Another available method focuses ona difference in the amount of change among signal components of the RGBsignal and separates color components of a face video into thewavelength and spectral intensity of reflected light to provide adetection method insusceptible to environmental changes(JP-2018-86130-A). The method is as described below. That is, the pulsewave receives pressure from the heart, and blood flow periodicallyvaries in response to heart beat. Light absorbs hemoglobin in the blood,and thus displacement of the spectral intensity is observed on the basisof a video corresponding to the skin under constant external light.However, the spectral intensity varies according to fluctuation inexternal light, often leading to noise. Thus, JP-2018-86130-A disclosesthat light is converted into a wavelength component and that fluctuationin spectral wavelength is detected as a pulse wave.

SUMMARY OF THE INVENTION

The above-described technique is expected as a measurement method inwhich no load is imposed on a subject. In these circumstances, atechnique for performing measurements in a non-contact manner from aremote position using a camera is important as a technique formonitoring a driver driving a vehicle or a sudden change in condition athome due to a respiratory disease, a heart failure, or the like.

A face video is formed by, for example, reflected light returning, whilescattering, from the skin into which illumination light radiated to theface has been absorbed. Thus, the known method using the RGB signalcorresponds to observation of variations in the spectral intensity ofreflected light in three colors. Thus, in a case where the face isexposed to steady light, the pulse can be stably detected. However,detection is precluded during night-time or in a dark room. Thus, in acertain method, a near-infrared camera replaces a visible-light cameraand radiates infrared light, and variations in infrared signal aremeasured instead of variations in RGB signal to detect the pulse wave.However, the infrared signal penetrates the skin deeper than the RGBsignal and is thus susceptible to noise. Additionally, in an infraredregion with no color information, identifying a skin area of the face isdifficult.

In view of these problems, an aspect of the present invention is todetect the pulse wave by measuring variations in the wavelength of aninfrared spectrum.

To accomplish the object, a typical aspect of the present inventionprovides a biological-information detection device including a videoinput section accepting video signals including three wavelengthcomponents in an infrared region included in reflected light from anobject, a wavelength detecting section acquiring a wavelength and anintensity of the reflected light from the video signals, a face featureamount detecting section detecting a plurality of feature points of aface on a basis of the video signals, a measurement target areaidentifying section identifying a measurement target area on a basis ofthe plurality of feature points of the face detected, a wavelengthfluctuation detecting section detecting a difference between awavelength of reflected light from the measurement target area at acertain point in time and a wavelength of reflected light at a point intime preceding the certain point in time, and a pulse wave detectingsection detecting, as a pulse wave, a change in the detected differenceaccording to the point in time.

According to an aspect of the present invention, monitoring can beperformed during night-time and the like in a non-contact manner.

The objects, configurations, and effects other than those describedabove will be apparent from the description of embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of a configuration ofa biological-information detecting device according to Embodiment 1;

FIG. 2 is a diagram illustrating an example of an IR camera module ofthe biological-information detecting device according to Embodiment 1;

FIG. 3 is a diagram illustrating an example of a video input section ofthe biological-information detecting device according to Embodiment 1;

FIG. 4A is a diagram illustrating an example of a wavelength signalgenerating section of the biological-information detecting deviceaccording to Embodiment 1;

FIG. 4B is a diagram illustrating an example of a wavelength signalgenerating section of the biological-information detecting deviceaccording to Embodiment 1, the wavelength signal generating sectionincluding a plurality of face feature amount detecting sections;

FIG. 5 is a diagram illustrating an example of a wavelength fluctuationdetecting section of the biological-information detecting deviceaccording to Embodiment 1;

FIG. 6 is a diagram illustrating an example of a pulse wave detectingsection of the biological-information detecting device according toEmbodiment 1;

FIG. 7A is a diagram illustrating a 3LCD system that is an opticalsystem in a projector, as an example of an optical system related to thebiological-information detecting device according to Embodiment 1;

FIG. 7B is a diagram illustrating an example of an optical system in thebiological-information detecting device according to Embodiment 1;

FIG. 8 is a diagram illustrating an example of a spatial filteraccording to Embodiment 1;

FIG. 9 is a diagram illustrating specified ranges of an HSV color spaceand a partial color space according to Embodiment 1;

FIG. 10A is a diagram illustrating an example of a band-pass filter of aCCD camera;

FIG. 10B is a diagram illustrating an example of an RGB filter of theCCD camera;

FIG. 10C is a diagram illustrating an example of an infrared filter ofan optical system in a biological-information detecting device accordingto Embodiment 2;

FIG. 11A is a diagram illustrating an example of a method for setting apartial color space according to Embodiment 1;

FIG. 11B is a diagram illustrating an example of a method for adjustingthe level of an IR camera signal according to Embodiment 1;

FIG. 12 is a diagram illustrating an example of a method for adjustingthe level of face area detection according to Embodiment 1;

FIG. 13 is a diagram illustrating an example of a skin area detected bya face feature amount detecting section of the biological-informationdetecting device according to Embodiment 1;

FIG. 14 is a block diagram illustrating an example of a configuration ofthe biological-information detecting device according to Embodiment 2;

FIG. 15 is a diagram illustrating an example of an IR camera module ofthe biological-information detecting device according to Embodiment 2;and

FIG. 16 is a diagram illustrating an example of spectral sensitivitiesof IR cameras and transmittances of filters, the IR cameras and thefilters being used in the biological-information detecting deviceaccording to Embodiment 1.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention will be described below on thebasis of the drawings. However, the present invention is not limited tothe embodiments. Note that, in the drawings illustrating theembodiments, the same members are denoted by the same reference signs,and repeated descriptions of the members are omitted.

Embodiment 1

In the present embodiment, an example of a biological-informationdetecting device will be described that functions to detect a pulse froma face video using an infrared (IR) light source and three cameras.

FIG. 1 is a block diagram illustrating an example of a configuration ofa biological-information detecting device according to Embodiment 1.

The biological-information detecting device according to the presentembodiment includes an IR1 to an IR3 cameras 200 a to 200 c that are afirst to a third IR cameras, respectively, video input sections 300 a to300 c, a wavelength signal generating section 400, a wavelengthfluctuation detecting section 500, a pulse wave detecting section 600,and a data display section 107.

Infrared imaging data, that is, an IR1 imaging data signal 101 a outputfrom the IR1 camera 200 a, is input to the video input section 300 a.IR2 imaging data signal 101 b output from the IR2 camera 200 b is inputto the video input section 300 b. IR3 imaging data signal 101 c outputfrom the IR3 camera 200 c is input to the video input section 300 c. Adelayed IR1 data signal 102 a, a delayed IR2 data signal 102 b, and adelayed IR3 data signal 102 c output from the video input sections 300 ato 300 c, respectively, are input to the wavelength signal generatingsection 400. A wavelength data signal 103 and a level signal 104 outputfrom the wavelength signal generating section 400 are input to thewavelength fluctuation detecting section 500. An average wavelengthdifference data signal 105 output from the wavelength fluctuationdetecting section 500 is input to the pulse wave detecting section 600.A pulse signal 106 output from the pulse wave detecting section 600 isinput to the data display section 107.

FIG. 2 is a diagram illustrating an example of an IR camera module ofthe biological-information detecting device according to Embodiment 1.

In the IR1 camera 200 a, a charge-coupled device (CCD) 202 is irradiatedwith infrared light through an IR1 transmitting filter 1 (band-passfilter or low pass filter) 201 a installed for a lens. The band of theIR1 transmitting filter 201 a installed at the lens is included in theband of an infrared transmitting and visible-light cut filter mounted inthe camera. The light is photoelectrically converted into an electricsignal 203 which is then digitized by an analog-to-digital (AD)converter 204. A resultant digital data signal 205 is input to a videocorrecting section 207.

The video correcting section 207 performs video adjustment such as whitebalancing and contrasting using a video adjustment parameter 206 a, andoutputs IR1 imaging data signal 101 a including an IR wavelength foreach pixel as a component.

The IR2 camera 200 b and the IR3 camera 200 c are configured similarlyto the IR1 camera 200 a except that the IR transmitting filter installedat a lens in each of the IR2 and IR3 cameras 200 b and 200 c is aband-pass filter or a low pass filter having a wavelength different fromthe band of the IR1 transmitting filter 201 a. In the description of thepresent configuration, the IR1 transmitting filter 201 a is installed atthe lens. However, the IR1 transmitting filter 201 a may be replacedwith a mounted infrared transmitting and visible-light cut filter andmay be installed between the lens and the CCD. The CCD camera has beendescribed by way of example. However, the CCD 202 is an example of alight receiving section (image sensor), and each of the IR1 to IR3cameras 200 a to 200 c may include a light receiving section other thanthe CCD (for example, indium gallium arsenide (InGaAs)).

The biological-information detecting device according to the presentembodiment includes three cameras. However, video adjustment parameters206 a, 206 b, and 206 c respectively provided in the IR1 camera 200 a,the IR2 camera 200 b, and the IR3 camera 200 c may be the same ordifferent as long as the IR1 camera 200 a includes the infraredtransmitting visible-light filter 201 a, the IR2 camera 200 b includesan infrared transmitting visible-light filter (not illustrated)different from that of the IR1 camera, and the IR3 camera 200 c includesan infrared transmitting visible-light filter (not illustrated)different from those of the IR1 and IR2 cameras. All the IR cameras arethe same in the remaining part of the configuration, and thus,illustration of detailed configurations of the IR2 camera 200 b and theIR3 camera 200 c is omitted.

FIG. 16 is a diagram illustrating an example of spectral sensitivitiesof the IR cameras and transmittances of the filters, the IR cameras andthe filters being used in the biological-information detecting deviceaccording to Embodiment 1.

The wavelength on the horizontal axis corresponds to near infrared raysand ranges from 700 to 1,000 nm, a solid graph indicates the spectralsensitivity of the CCD, and the ordinate axis indicates a quantumefficiency. Additionally, dashed graphs indicate characteristic curvesof three types of band-pass filters, with the transmittance of thefilter indicated on the ordinate axis. FIG. 16 illustrates IR1, IR2, andIR3 in order of increasing wavelength. However, the order and thebandwidths may be changed and a low pass filter may be used.

Additionally, the IR1 camera, the IR2 camera, and the IR3 camera need tocapture an image of the same subject, and thus, the three cameras may bearranged in juxtaposition and videos may be geometrically corrected tothe same position. Alternatively, correction may be made to allow imagesto be captured at optically the same position.

FIG. 7A is a diagram illustrating a 3 liquid crystal display (LCD)method that is one of optical systems in projectors, as an example of anoptical system according to the biological-information detecting deviceaccording to Embodiment 1.

A light ray output from a light source is split into a red wavelengthR701, a green wavelength G702, and a blue wavelength B703 which areinput to a 3LCD optical system 700. In the optical system 700, lightrays respectively pass through a red LCD 704, a green LCD 705, and ablue LCD 706 to generate a video with an R component, a video with a Gcomponent, and a video with a B component. The light rays constitutingthe videos are transmitted and reflected by dichroic prisms asillustrated in FIG. 7A to provide a synthesized video. The video passesthrough a lens and is formed into a video on a screen.

FIG. 7B is a diagram illustrating an example of an optical system in thebiological-information detecting device according to Embodiment 1.

Specifically, FIG. 7B illustrates a method for acquiring a video bytracking an optical path opposite to the optical path in the 3LCD methodillustrated in FIG. 7A. As illustrated in an optical system 750 in FIG.7B, a video is converted into an R component, a G component, and a Bcomponent through a lens and dispersed through dichroic prisms thattransmit and reflect IR1 (751), IR2 (752), and IR3 (753). Resultantlight rays pass through the IR1 transmitting filter 201 a, the IR2transmitting filter 201 b, and the IR3 transmitting filter 201 c to theIR cameras 200 a, 200 b, and 200 c in which the light rays are convertedinto videos with the respective components. Additionally, with anexternal light source such as an incandescent lamp, infrared raysemitted from the light source may be utilized. With no light source,infrared light including infrared wavelengths IR1, IR2, and IR3 may beseparately prepared.

FIG. 3 is a diagram illustrating an example of the video input sectionof the biological-information detecting device according to Embodiment1.

The video input section 300 a includes a video acquiring section 301 anda video data storage section 303. The video acquiring section 301receives the IR1 imaging data signal 101 a as an input signal andoutputs an IR data signal 302 for a video. The video data storagesection 303 receives the IR data signal 302 for one frame as an inputsignal and outputs the delayed IR1 data signal 102 a. The video inputsection 300 b and the video input section 300 c are configured similarlyto the video input section 300 a, and thus, illustration of the videoinput sections 300 b and 300 c is omitted.

FIG. 4A is a diagram illustrating an example of the wavelength signalgenerating section 400 of the biological-information detecting deviceaccording to Embodiment 1.

The wavelength signal generating section 400 includes spatial filters401 a, 401 b, and 401 c, a wavelength detecting section 403, a facefeature amount detecting section 405 a, a skin area estimating section408, and a switch 409. The wavelength signal generating section 400executes video processing for each pixel.

The spatial filters 401 a, 401 b, and 401 c receive, as input signals,the delayed IR1 data signal 102 a, the delayed IR2 data signal 102 b,and the delayed IR3 data signal 102 c, respectively, each including aline delay corresponding to taps of a convolution kernel. For example,the spatial filters 401 a, 401 b, and 401 c perform weighted averagingon pixels around a pixel of interest and outputs a smoothed delayed IR1data signal 402 a, a smoothed delayed IR2 data signal 402 b, and asmoothed delayed IR3 data signal 402 c.

The wavelength detecting section 403 receives, as input signals, thesmoothed delayed IR1 data signal 402 a, the smoothed delayed IR2 datasignal 402 b, and the smoothed delayed IR3 data signal 402 c, andconverts the data signals into the wavelength data signal 103.

The face feature amount detecting section 405 a, for example, receives,as an input signal, the delayed IR1 data signal 102 a, one of thedelayed IR data signals, and adjusts the sharpness, contrast, and noiseof the IR video according to the wavelength of the IR1 transmittingfilter 201 a provided in the IR1 camera 200 a, on the basis of an IR1video adjustment parameter 404 a. The face feature amount detectingsection 405 a outputs an IR1 face feature amount 406 a such as thecoordinates and sizes of the eyes, the mouth, and the nose.

The skin area estimating section 408 functions to identify a measurementtarget area for the biological-information detecting device.Specifically, the skin area estimating section 408 receives the IR1 facefeature amount 406 a as an input signal and estimates a skin area usedfor pulse wave detection, for example, on the basis of position offsetsset in accordance with a skin area estimating section parameter 407. Theskin area estimating section 408 outputs the level signal 104 indicatingthe skin area. A switch 409 enables or disables the skin area estimatingfunction in a switchable manner. In the present embodiment, the thusestimated skin area is identified as a measurement target area.

In this case, the wavelength detecting section 403 may set the smootheddelayed IR1 data signal 402 a, the smoothed delayed IR2 data signal 402b, and the smoothed delayed IR3 data signal 402 c, for example, as R, G,and B and use an RGB-HSV conversion. The wavelength detecting section403 may output H (Hue) as the wavelength data signal 103.

FIG. 13 is a diagram illustrating an example of the skin area detectedby the face feature amount detecting section 405 a of thebiological-information detecting device according to Embodiment 1.

Examples of feature parts constituting the face include the eyes, themouth, the nose, the eyebrows, and the ears. The eyes and the mouth areused in the description of the present embodiment.

For example, the face feature amounts of the eyes and the mouth can beidentified using a feature classifier such as Haar-like or a localbinary pattern (LBP) as is the case with detected face feature areas inFIG. 13, and can be calculated, for example, as rectangular areas. Pulsewave detection in the present embodiment utilizes changes in the colorof the skin and thus needs detection of skin areas. Thus, the facefeature amount detecting section 405 a estimates, as a skin area of theface, a rectangle located at an upper, a lower, a left, and a rightoffset distances in FIG. 13, for example, from the rectangular positionsof the eyes and the mouth.

In this case, the skin area needs to be located inside the face, andthus, the face feature amount detecting section 405 a performs facedetection and determines whether the skin area is included in therectangle of the detected face. In a case where the skin area is notincluded in the rectangle, the face feature amount detecting section 405a invalidates the skin area by, for example, setting the size of theskin area to zero.

As described above, the skin area is estimated on the basis of thefeatures of the face such as the eyes and the mouth to allow themeasurement target area to be identified on the basis of signals in theinfrared region with no color information. Additionally, appropriateadjustment of the offsets allows the accuracy of estimation of the skinarea to be improved.

FIG. 4B is a diagram illustrating an example of the wavelength signalgenerating section of the biological-information detecting deviceaccording to Embodiment 1, the wavelength signal generating sectionincluding a plurality of face feature amount detecting sections 405 a to405 c.

In FIG. 1, the wavelength signal generating section 400 may be replacedwith a wavelength signal generating section 450 illustrated in FIG. 4B.The wavelength signal generating section 450 includes the spatialfilters 401 a, 401 b, and 401 c, the wavelength detecting section 403,the face feature amount detecting sections 405 a, 405 b, and 405 c, thefeature amount adjusting section 451, the skin area estimating section408, and the switch 409.

The spatial filters 401 a, 401 b, and 401 c receive, as input signals,the delayed IR1 data signal 102 a, the delayed IR2 data signal 102 b,the delayed IR3 data signal 102 c, and outputs the smoothed delayed IR1data signal 402 a, the smoothed delayed IR2 data signal 402 b, and thesmoothed delayed IR3 data signal 402 c. The wavelength detecting section403 receives, as input signals, the smoothed delayed IR1 data signal 402a, the smoothed delayed IR2 data signal 402 b, and the smoothed delayedIR3 data signal 402 c, and converts the data signals into the wavelengthdata signal 103.

Additionally, the face feature amount detecting section 405 a receivesthe delayed IR1 data signal 102 a and the IR1 video adjustment parameter404 a as input signals, and outputs the IR1 face feature amount 406 a.The face feature amount detecting section 405 b receives the delayed IR2data signal 102 b and the IR2 video adjustment parameter 404 b as inputsignals, and outputs an IR2 face feature amount 406 b. The face featureamount detecting section 405 c receives the delayed IR3 data signal 102c and the IR3 video adjustment parameter 404 c as input signals, andoutputs an IR3 face feature amount 406 c.

The feature amount adjusting section 451 receives the IR1 face featureamount 406 a, the IR2 face feature amount 406 b, and the IR3 facefeature amount 406 c as input signals, and averages the input signals,that is, numerical values such as coordinates and sizes. The featureamount adjusting section 451 outputs an adjusted face feature amount452.

The skin area estimating section 408 estimates the skin area used forpulse wave detection, for example, on the basis of position offsets setin accordance with the skin area estimating section parameter 407, andoutputs the level signal 104 indicating the skin area. The switch 409enables or disables the skin area estimating function in a switchablemanner.

In this case, the calculation by the feature amount adjusting section451 is, for example, the calculation of the average value of the IR facefeature amounts, but may be calculation of an intermediate value.Additionally, in a case where any face feature amount fails to bedetected, the feature amount may be excluded from the calculation ordata output of the adjusted face feature amount may be omitted or theadjusted face feature amount may be set to zero.

When signals with three infrared wavelengths are acquired as describedabove, any of the signals may fail to be used due to the effect of noiseor the like. In such a case, the effect of noise included in theinfrared signal can be reduced by excluding the unusable signal,detecting the face feature amounts in usable signals, and estimating theskin area on the basis of the average value, intermediate value, or thelike of the detected face feature amounts.

As described above, the wavelength signal generating section 400illustrated in FIG. 4A or the wavelength signal generating section 450illustrated in FIG. 4B can convert displacement of the spectralintensity into displacement of the wavelength.

FIG. 12 is a diagram illustrating an example of a method for adjustingthe level of face area detection according to Embodiment 1.

A face area detection switch is operatively associated with the switch409 which enables or disables the skin area estimating function in aswitchable manner. The wavelength signal generating section 400 or 450may output a signal from the skin area estimating section 408 as thelevel signal 104 indicating the skin area when the face area detectionswitch is on, and may select the entire area as the skin area when theface area detection switch is off. For example, assume that the skinarea in the video is 1 and any other area in the video is 0, the levelsignal 104 indicating the skin area may be forced to be 1 regardless ofthe output from the skin area estimating section 408 in a case where theface area detection switch is off.

FIG. 5 is a diagram illustrating an example of the wavelengthfluctuation detecting section 500 of the biological-informationdetecting device according to Embodiment 1.

The wavelength fluctuation detecting section 500 includes a wavelengthdata storage section 501, a wavelength difference calculating section503, a wavelength difference integrating section 505, a skin areacalculating section 507, and an average-wavelength-differencecalculating section 509. The wavelength data storage section 501 storesthe wavelength data signal 103 for each frame and outputs a delayedwavelength data signal 502 resulting from frame delay.

The wavelength difference calculating section 503 receives, as input,the level signal 104 indicating the skin area, the wavelength datasignal 103, and the delayed wavelength data signal 502. In a case wherea signal for pixels in the skin area is input to the wavelengthdifference calculating section 503 (that is, 1 is input as the levelsignal 104), the wavelength difference calculating section 503 outputs awavelength difference data signal 504 calculated from the inputwavelength data signal 103 and delayed wavelength data signal 502 (thatis, the wavelength difference data signal 504 is a difference betweenthe wavelength data signal 103 at a certain point in time and thewavelength data signal 103 at a point in time preceding the certainpoint in time). In a case where a signal for pixels other than thepixels in the skin area is input to the wavelength differencecalculating section 503, the wavelength difference calculating section503 outputs a 0 value.

The wavelength difference integrating section 505 receives, as input,the wavelength difference data signal 504 for the pixels in the skinarea, integrates wavelength differences for respective frames, andoutputs an integrated wavelength difference data signal 506. The skinarea calculating section 507 receives, as input, the level signal 104indicating the skin area, counts the number of the pixels in the skinarea for each frame, and outputs a skin area signal 508. Theaverage-wavelength-difference calculating section 509 receives, asinput, the integrated wavelength difference data signal 506 and skinarea signal 508 for the skin area pixels, calculates the wavelengthdifference for each frame, and outputs the average wavelength differencedata signal 105.

FIG. 6 is a diagram illustrating an example of the pulse wave detectingsection 600 of the biological-information detecting device according toEmbodiment 1.

The pulse wave detecting section 600 includes a difference data storagesection 601, a smoothing filter 603, a smoothed data storage section605, an inclination detecting section 607, a sign data storage section609, and an extremum detecting section 611, and executes videoprocessing for each frame.

The difference data storage section 601 receives the average wavelengthdifference data signal 105 as input and outputs a delayed wavelengthdifference data signal 602. The smoothing filter 603 receives theaverage wavelength difference data signal 105 and the delayed wavelengthdifference data signal 602 as input and outputs a wavelength differencedata signal 604 resulting from smoothing, on a continuous time axis, ofwavelength data for a plurality of frames.

The smoothed data storage section 605 receives the smoothed wavelengthdifference data signal 604 as input, holds wavelength difference datafor a plurality of frames, and outputs a smoothed delayed wavelengthdifference data signal 606. The inclination detecting section 607determines a difference between two consecutive frame data or betweenaverage frames among several consecutive neighbor frames, and outputs asign data signal 608 indicating the sign of an inclination.

The sign data storage section 609 receives the sign data signal 608 asinput, holds sign data for a plurality of frames, and outputs a delayedsign data signal 610. The extremum detecting section 611 receives thesign data signal 608 and the delayed sign data signal 610 as input, anddetermines extremums by setting, as a maximal value, a frame having aninclination with a sign having changed from a positive value to anegative value and setting, as a minimal value, a frame having aninclination with a sign having changed from a negative value to apositive value. The extremum detecting section 611 outputs, for example,the maximal value as a pulse signal 106.

FIG. 8 is a diagram illustrating an example of a spatial filteraccording to Embodiment 1.

FIG. 8 is an example in which a matrix of 3×3 taps, that is, a 3×3convolution kernel is applied to an image. Values obtained by performinga convolution operation on the kernel around a pixel of interest in theimage are the smoothed delayed IR1 data signal 402 a, the smootheddelayed IR2 data signal 402 b, and the smoothed delayed IR3 data signal402 c. Values in the kernel are weighted average coefficients, and thesum of the values may be 1.0. For example, a mean value distribution, aGaussian distribution, or the like can be used for smoothing.

FIG. 9 is a diagram illustrating an example of specified ranges of anHSV color space and a partial color space according to Embodiment 1.

FIG. 9 expresses the HSV color space in cylindrical coordinates. Thevertical axis corresponds to Value, that is, brightness, and indicatesthe lightness of colors. The axis in the radial direction corresponds toSaturation and indicates the density of colors. The rotation anglecorresponds to Hue. The hue is independent of intensity and density, andis considered to correspond to a wavelength component of reflected lightgiven that imaging captures reflection of light. Similarly, thebrightness can be considered to indicate the intensity of a particularwavelength.

The color space for the visible light region has been described.However, the visible light region is replaced with the infrared regionfor the biological-information detecting device of the presentembodiment. For example, R, G, and B in the color space may berespectively replaced with IR1, IR2, and IR3, and an RGB-HSV conversionmay be used to convert the IR space into the HSV space. The HSV space isused in the above description. However, a hue-saturation-lightness (HSL)space may be used or any other color space may be used as long as thespace allows conversions into components similar to wavelengthcomponents.

FIG. 11A is a diagram illustrating an example of a method for settingthe partial color space according to Embodiment 1.

For example, the data display section 107 of the biological-informationdetecting device may display bars and icons; as illustrated in FIG. 11A,the bars indicate the entire ranges of the wavelength, deviation, andintensity, and the icons indicate both ends (for example, “Infrared 1”and “Infrared 2” specifying the wavelength range) of the range specifiedon each of the bars. Here, the wavelength, the deviation, and theintensity respectively correspond to the hue, the saturation, and thebrightness in the color space for the infrared region. The user canspecify the range by using an input device (not illustrated) of thebiological-information detecting device to operate the correspondingicons.

For example, for the wavelength range, an angle of 0 degrees=360 degreesin the wavelength space corresponds to IR1, and angles of 120 degreesand 240 degrees in the wavelength space respectively correspond to IR2and IR3. For example, the range may be specified as a section definedusing Infrared 1 and Infrared 2. For the deviation range, 0% correspondsto even infrared rays, and 100% corresponds to the maximum deviation ofinfrared rays. For example, the range may be specified to lie fromDeviation 1 to Deviation 2. The intensity ranges from 0% to 100%, andthe range may similarly be specified to lie from Intensity 1 toIntensity 2.

Note that the wavelength range, the deviation range, and the intensityrange may be manually set as illustrated in FIG. 11A but may beautomatically set. Now, an example of a method for automatic settingwill be described.

For example, the skin area estimating section 408 estimates the skinarea as illustrated in FIG. 13 on the basis of the face feature amountdetected by at least one of the face feature amount detecting sections405 a to 405 c. Subsequently, the skin area estimating section 408 mayset the wavelength range, the deviation range, and the intensity rangesuch that the wavelength range, the deviation range, and the intensityrange include the wavelengths, deviations, and intensities of at leastsome of the pixels in the skin area estimated on the basis of the facefeature amount. For example, on the basis of appearance frequencydistributions of the wavelength, deviation, and intensity values of thepixels in the skin area, the skin area estimating section 408 may set,as the wavelength range, the deviation range, and the intensity range,the ranges of the wavelength, deviation, and intensity in which theappearance frequency is higher than a predetermined value.

Subsequently, the skin area estimating section 408 estimates the skinarea on the basis of a partial space (see FIG. 9) identified by the setwavelength range, deviation range, and intensity range. For example, theskin area estimating section 408 may finally estimate, as a skin area,that area of the pixels in the skin area estimated on the basis of theface feature amount in which the wavelength, deviation, and intensityvalues are respectively included in the above-described wavelengthrange, deviation range, and intensity range. Alternatively, the skinarea estimating section 408 estimate, as a skin area, that area of thepixels in which the wavelength, deviation, and intensity values arerespectively included in the above-described wavelength range, deviationrange, and intensity range, regardless of the skin area estimated on thebasis of the face feature amount.

In the present embodiment, the skin area can be estimated by utilizingthe face feature amount. However, the position of the actual skin areamay be moved, for example, the subject may change the posture as thetime elapses. As a result, the estimated skin area includes an areaother than the skin area, for example, the eyes or the mouth, which maylower the accuracy of detection of the pulse wave. In order to followmovement of the actual skin area, the face feature amount may constantlybe detected, and on the basis of the face feature amount, the skin areamay be estimated. However, detection of the face feature amount requireshigh calculation costs.

In contrast, the wavelength, deviation, and intensity values of pixelsin the area other than the skin area such as the eyes and the mouth areexpected to be out of the wavelength range, the deviation range, and theintensity range set as described above. Additionally, estimation of theskin area based on the partial space requires low calculation costs.Thus, the combination of estimations based on the partial space asdescribed above enables the pulse wave to be accurately detected using asmall amount of calculation while eliminating noise attributed to thearea other than the skin area.

FIG. 11B is a diagram illustrating an example of a method for adjustingthe level of an IR camera signal according to Embodiment 1.

For example, the data display section 107 of the biological-informationdetecting device may display bars and icons; as illustrated in FIG. 11B,the bars indicate the entire ranges (from 0% to 100%) of a signal level1, a signal level 2, and a signal level 3, and each of the iconsindicates a signal level on the corresponding bar. In this case, thesignal level 1 corresponds to the video adjustment parameter 206 a setfor the IR1 camera 200 a. Similarly, the signal level 2 and the signallevel 3 correspond to the video adjustment parameters (not illustrated)set for the IR2 camera 200 b and the IR3 camera 200 c. By using theinput device (not illustrated) of the biological-information detectingdevice to operate the icons, the user can adjust the output from thewavelength detecting section 403.

The above-described configuration enables monitoring that suppresses thenegative effect of noise in a non-contact manner and that detects, evenduring nighttime, a sudden change in condition due to a disease, a heartfailure, or the like.

Embodiment 2

In Embodiment 1, the technique has been described that uses the IR lightsource and the three cameras to suppress the negative effect of noiseand to detect a pulse from the face video. In Embodiment 2, abiological-information detecting device using one camera formeasurements will be described. The sections of thebiological-information detecting device of Embodiment 2 include the samefunctions as those of the sections with the same reference signs inEmbodiment 1 illustrated in FIGS. 1 to 9, FIGS. 11A to 13, FIG. 16, andthe like except for differences described below. Descriptions of thesesections are omitted.

FIG. 14 is a block diagram illustrating an example of a configuration ofthe biological-information detecting device according to Embodiment 2.

The biological-information detecting device according to the presentembodiment includes an IR camera 200 d, a video input section 300 d, thewavelength signal generating section 400, the wavelength fluctuationdetecting section 500, the pulse wave detecting section 600, and thedata display section 107.

The IR camera 200 d detects each of the three infrared components, andsynthesizes the three components into a pack signal, that is, an IRimaging data signal 101 d. The IR camera 200 d outputs the IR imagingdata signal 101 d. The video input section 300 d receives the IR imagingdata signal 101 d as an input signal and decomposes a delayed IR datasignal 102 d into components. The video input section 300 d outputsresultant unpack signals, that is, the delayed IR data signals 102 a,102 b, and 102 c.

The wavelength signal generating section 400 receives the delayed IRdata signals 102 a, 102 b, and 102 c as input signals and outputs thewavelength data signal 103 and the level signal 104. The wavelengthfluctuation detecting section 500 receives the wavelength data signal103 and the level signal 104 as input signals, and outputs the averagewavelength difference data signal 105. The pulse wave detecting section600 receives the average wavelength difference data signal 105 as aninput signal, and outputs the pulse signal 106. The data display section107 receives the pulse signal 106 as an input signal, and displays data.Note that, in the present configuration, the wavelength signalgenerating section 400 may be replaced with the wavelength signalgenerating section 450.

FIG. 15 is a diagram illustrating an example of an IR camera module ofthe biological-information detecting device according to Embodiment 2.

In the IR camera 200 d, light having passed through the lens travelsthrough an IR transmitting filter (band-pass filter or low pass filter)201 d for three infrared wavelength bands. Infrared light with threewavelengths is radiated to the CCD 202. The light is photoelectricallyconverted into the electric signal 203, which is then digitized by theAD converter 204. The digital data signal 205 output from the ADconverter 204 is input to the video correcting section 207. The videocorrecting section 207 performs, on the input digital data signal 205,video adjustment such as white balancing and contrasting based on avideo adjustment parameter 206 d, and outputs, for example, an IRimaging data signal 101 d with an IR wavelength for each pixel as acomponent.

FIG. 10A is a diagram illustrating an example of the band-pass filter ofthe CCD camera.

As illustrated in FIG. 10A, the band-pass filter includes, for example,a color filter located above a CCD so as to overlap the CCD.Additionally, light receiving elements are provided on the CCD, and onelight receiving element corresponds to one cell of the color (RGB)filter, corresponding to a pixel.

FIG. 10B is a diagram illustrating an example of the RGB filter of theCCD camera.

One cell of the RGB filter is assigned with one color (that is, one ofthe R, G, and B), and the cells to which the respective colors areassigned are arranged in an array as illustrated in FIG. 10B. This isreferred to as a Bayer arrangement. Two other color components absentfrom one pixel are obtained by interpolating adjacent peripheral colors.

FIG. 10C is a diagram illustrating an example of an infrared filter ofan optical system in the biological-information detecting deviceaccording to Embodiment 2.

The infrared filter illustrated in FIG. 10C corresponds to the IR1transmitting filters, the IR2 transmitting filters, and the IR3transmitting filters replacing the transmitting filters of the RGBfilter in FIG. 10B, and represents the IR transmitting filter 201 d inFIG. 15.

The configuration of Embodiment 1 eliminates the need for signalinterpolation, but instead requires three cameras. In contrast, by usingone camera, the configuration of Embodiment 2 enables monitoring thatsuppresses the negative effect of noise in a non-contact manner and thatdetects, even during night-time, a sudden change in condition due to adisease, a heart failure, or the like.

Note that the present invention is not limited to the above-describedembodiments and includes various modified examples. For example, theabove-described embodiments have been described in detail for betterunderstanding of the present invention and are not limited to theembodiments including all of the described components. A part of theconfiguration of one embodiment can be replaced with the configurationof another embodiment, and the configuration of one embodiment can beadded to the configuration of another embodiment. A configuration can beadded to, removed from, or replace a part of the configuration of eachembodiment.

A part or all of each of the above-described configurations, functions,processing sections, processing means, and the like may be implementedin hardware by, for example, being designed in an integrated circuit.Each of the above-described configurations, functions, and the like maybe implemented in software by a processor interpreting and executingprograms realizing the respective functions. Information such asprograms, tables, and files used to implement the functions can bestored in a storage device such as a nonvolatile semiconductor memory, ahard disk drive, or a solid state drive (SSD), or a computer readablenon-transitory data storage medium such as an integrated circuit (IC)card, a secure digital (SD) card, or a digital versatile disc (DVD).

Control lines and information lines illustrated are considered to benecessary for description, and not all of the control lines andinformation lines for products are illustrated. In actuality,substantially all components may be considered to be connected together.

What is claimed is:
 1. A biological-information detection devicecomprising: a video input section accepting video signals includingthree wavelength components in an infrared region included in reflectedlight from an object; a wavelength detecting section acquiring awavelength and an intensity of the reflected light from the videosignals; a face feature amount detecting section detecting a pluralityof feature points of a face on a basis of the video signals; ameasurement target area identifying section identifying a measurementtarget area on a basis of the plurality of feature points of the facedetected; a wavelength fluctuation detecting section detecting adifference between a wavelength of reflected light from the measurementtarget area at a certain point in time and a wavelength of reflectedlight at a point in time preceding the certain point in time; and apulse wave detecting section detecting, as a pulse wave, a change in thedetected difference according to the point in time.
 2. Thebiological-information detecting device according to claim 1, whereinthe face feature amount detecting section detects eyes and a mouth asthe plurality of feature points of the face, and the measurement targetarea identifying section estimates a skin area on a basis of offsetsfrom the plurality of feature points, and identifies the estimated skinarea as the measurement target area.
 3. The biological-informationdetecting device according to claim 1, wherein the face feature amountdetecting section detects positions and sizes of the plurality offeature points on a basis of the video signals with the three wavelengthcomponents, and the measurement target area identifying sectionidentifies the measurement target area on a basis of average values orintermediate values of the positions and sizes of the feature pointsdetected on the basis of the video signals with the three wavelengthcomponents.
 4. The biological-information detecting device according toclaim 3, wherein in a case where the measurement target area identifyingsection fails to detect at least one of the positions and sizes of theplurality of feature points on the basis of one of the video signalswith the three wavelength components, the measurement target areaidentifying section identifies the measurement target area on a basis ofaverage values or intermediate values of the positions and sizes of thefeature points detected on a basis of the video signals with theremainder of the three wavelength components.
 5. Thebiological-information detecting device according to claim 1, whereinthe wavelength detecting section acquires wavelengths and intensities ina color space in which the three wavelengths in the infrared region arereplaced with red, green, and blue in a visible light region.
 6. Thebiological-information detecting device according to claim 5, whereinthe color space is a hue-saturation-value space or ahue-saturation-lightness space.
 7. The biological-information detectingdevice according to claim 5, wherein after identifying the measurementtarget area on the basis of the plurality of feature points of the face,the measurement target area identifying section identifies a partialspace included in the color space and including values of at least someof pixels in the identified measurement target area, and identifies anarea of the pixels in the partial space as the measurement target area.8. The biological-information detecting device according to claim 1,further comprising: three infrared cameras including infraredtransmitting filters with different spectral characteristics, whereinthe video input section accepts, from the three infrared cameras, videosignals including the three wavelength components.
 9. Thebiological-information detecting device according to claim 1, furthercomprising: an infrared camera including a light receiving sectionprovided with one of three types of infrared transmitting filters withdifferent spectral characteristics, wherein the video input sectionaccepts, from the infrared camera, video signals including the threewavelength components.
 10. A biological-information detection methodexecuted by a biological-information detecting device, comprising:accepting video signals including three wavelength components in aninfrared region included in reflected light from an object; acquiring awavelength and an intensity of the reflected light from the videosignals; detecting a plurality of feature points of a face on a basis ofthe video signals; identifying a measurement target area on a basis ofthe plurality of feature points of the face detected; detecting adifference between a wavelength of reflected light from the measurementtarget area at a certain point in time and a wavelength of reflectedlight at a point in time preceding the certain point in time; anddetecting, as a pulse wave, a change in the detected differenceaccording to the point in time.